Skip to content

Enable prefetching of SW kernel instructions after the first SW task #199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: develop
Choose a base branch
from
Open

Enable prefetching of SW kernel instructions after the first SW task #199

wants to merge 5 commits into from

Conversation

@Kepontry
Copy link

@Kepontry Kepontry commented Nov 7, 2025 •
edited by DariaMityagina
Loading

Summary

This PR enhances the AddSwKernelInstructionPrefetchPass to enable prefetching of SHAVE kernel instructions after the first SHAVE task, if the initial slack is insufficient.

Currently, instruction prefetching is skipped if the slack before the first SHAVE task is insufficient. This limitation is suboptimal when initial insertion slots (tiles) are limited or L2 cache capacity is constrained.

Based on the observation that SHAVE utilization is often low, we propose this change to prefetch opportunistically later in the schedule. This approach has demonstrated a ~3% performance gain on models such as Qwen2-1.5b and Qwen3-0.6b.

Target Platform For Release Notes

  • NPU37XX
  • NPU40XX
  • NONE (Not included in release notes)

Classification of this Pull Request

  • Maintenance
  • BUG
  • Feature

Implementation Details

  • The new logic searches for insertion gaps that begin at a "non-saturated" point (where num_shave_tasks < available_shave_count).
  • The gap ends at either a "saturated" point or the kernel designated for prefetching.
  • The prefetch operation is inserted at the tile3 task of the identified insertion point.
  • The minimal insertion slack required is set to 50K cycles.

Additional Fixes & Enhancements

  • Corrected an issue in insertDummyKernelOpBeforeFirstKernelTask where the clusterIdx was not being used during tile assignment.
  • Expanded Prefetching: Enriched the "kernel kind" logic to allow more types of kernels to be prefetched.

We also noted that the previous 250K-cycle threshold is overly conservative for our platform (Ultra 258V). Our analysis shows that prefetching provides benefits even with a slack as low as 170K cycles.

@Kepontry Kepontry requested a review from a team as a code owner November 7, 2025 08:18
@DariaMityagina
Copy link
Contributor

DariaMityagina commented Nov 7, 2025 •
edited
Loading

@Kepontry hello! Thanks for your PR!

Could you please ensure your changes include test coverage by adding tests to https://github.com/openvinotoolkit/npu_compiler/blob/develop/tests/lit/NPU/dialect/VPUIP/passes/add_sw_kernel_instruction_prefetch_40XX.mlir and maybe some functional tests?

ksenia-shkileva
ksenia-shkileva reviewed Nov 7, 2025
View reviewed changes
@DariaMityagina
Copy link
Contributor

DariaMityagina commented Nov 10, 2025

@Kepontry hello! Thanks for your PR!

Could you please ensure your changes include test coverage by adding tests to https://github.com/openvinotoolkit/npu_compiler/blob/develop/tests/lit/NPU/dialect/VPUIP/passes/add_sw_kernel_instruction_prefetch_40XX.mlir and maybe some functional tests?

@Kepontry could you please look into this comment? Thank you!

@Kepontry
Copy link
Author

Kepontry commented Nov 19, 2025

Apologies for the delay; I missed the email notification for this thread. I am currently working on adding the test. Could you provide some guidance or documentation on how to use the lit test framework within the NPU compiler?

@Kepontry
Copy link
Author

Kepontry commented Nov 19, 2025

Functional test added.

@Maxim-Doronin
Copy link
Collaborator

Maxim-Doronin commented Nov 24, 2025

Hi @Kepontry! Please adhere to the clang-format guidelines. You will find the automatically fixed code style in the job logs: https://github.com/openvinotoolkit/npu_compiler/actions/runs/19637113134/job/56230572067?pr=199

I also noticed that some LIT tests failed. Could you please verify if these failures are due to your changes?
https://github.com/openvinotoolkit/npu_compiler/actions/runs/19632108154/job/56223833881

cc @DariaMityagina

@Kepontry
Copy link
Author

Kepontry commented Nov 24, 2025

Hi @Maxim-Doronin , the failure of LIT test is caused by the DummySWKernelsForInstructionPrefetchReservedMemory not being found. I can reproduce this error by setting the minimum-shave-start-time-for-prefetch threshold to 5 in the default_hw_mode_40XX test. So the problem exists before this PR. I suspect that the createSWKernelInstructionPrefetchReserveMemForDummyKernelsPass function in the VPU pipeline is not called, but I am not entirely sure. I would appreciate your help verifying this.

vpux-opt --split-input-file --init-compiler="vpu-arch=NPU40XX compilation-mode=DefaultHW allow-custom-values=true" --mlir-elide-elementsattrs-if-larger 8 --default-hw-mode-vpuip="function-outlining='naive'" --add-sw-kernel-instruction-prefetch="minimum-shave-start-time-for-prefetch=5" default_hw_mode_40XX.mlir

@DariaMityagina
Copy link
Contributor

DariaMityagina commented Nov 25, 2025

Hi @Maxim-Doronin , the failure of LIT test is caused by the DummySWKernelsForInstructionPrefetchReservedMemory not being found. I can reproduce this error by setting the minimum-shave-start-time-for-prefetch threshold to 5 in the default_hw_mode_40XX test. So the problem exists before this PR. I suspect that the createSWKernelInstructionPrefetchReserveMemForDummyKernelsPass function in the VPU pipeline is not called, but I am not entirely sure. I would appreciate your help verifying this.

vpux-opt --split-input-file --init-compiler="vpu-arch=NPU40XX compilation-mode=DefaultHW allow-custom-values=true" --mlir-elide-elementsattrs-if-larger 8 --default-hw-mode-vpuip="function-outlining='naive'" --add-sw-kernel-instruction-prefetch="minimum-shave-start-time-for-prefetch=5" default_hw_mode_40XX.mlir

Hello @Kepontry!

Thanks for adding the tests and sharing your findings regarding pre-commit failures!
I'll check them locally and get back to you.

@DariaMityagina
Copy link
Contributor

DariaMityagina commented Nov 27, 2025 •
edited
Loading

@Kepontry I managed to reproduce the issue:

vpux-opt --split-input-file --init-compiler="vpu-arch=NPU40XX compilation-mode=DefaultHW allow-custom-values=true" --mlir-elide-elementsattrs-if-larger 8 --default-hw-mode-vpuip="function-outlining='naive'" --add-sw-kernel-instruction-prefetch="minimum-shave-start-time-for-prefetch=5" default_hw_mode_40XX.mlir

->

Cannot find DummySWKernelsForInstructionPrefetchReservedMemory!

Will research it a bit and get back!

In the meantime, could you please share with us why you set this particular value?
--add-sw-kernel-instruction-prefetch="minimum-shave-start-time-for-prefetch=5"

@Kepontry
Copy link
Author

Kepontry commented Nov 28, 2025

Hi, @DariaMityagina , thanks for your assistance regarding this issue. Since this PR enables prefetching regardless of the first SHAVE task's start time threshold, it exposed some existing bugs in certain test cases. These bugs were previously hidden because there wasn't enough time slack to trigger the prefetch logic. I adjusted the threshold to simulate a scenario that forces prefetch insertion, confirming that these test cases fail without this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ksenia-shkileva ksenia-shkileva ksenia-shkileva left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

READY_FOR_REVIEW

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Kepontry @DariaMityagina @Maxim-Doronin @ksenia-shkileva